Found while fuzzing m-c 20230201-b7f075124503 (--enable-address-sanitizer --enable-fuzzing)

Requires pref image.avif.sequence.enabled=true

==609797==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x62e00002e043 at pc 0x7ff9d86c0aeb bp 0x7ff9a99f9b10 sp 0x7ff9a99f9b08
READ of size 1 at 0x62e00002e043 thread T21
    #0 0x7ff9d86c0aea in qcms::transform_avx::qcms_transform_data_template_lut_avx::ha00637895f9bfafa /builds/worker/checkouts/gecko/gfx/qcms/src/transform_avx.rs:116:22
    #1 0x7ff9d86c0aea in qcms_transform_data_rgba_out_lut_avx /builds/worker/checkouts/gecko/gfx/qcms/src/transform_avx.rs:219:5
    #2 0x7ff9cb236165 in mozilla::image::ColorManagementFilter<mozilla::image::BlendAnimationFilter<mozilla::image::SurfaceSink>>::DoAdvanceRowFromBuffer(unsigned char const*) /builds/worker/checkouts/gecko/image/SurfaceFilters.h:162:5
    #3 0x7ff9cb1bf7ac in AdvanceRow /builds/worker/checkouts/gecko/image/SurfacePipe.h:141:19
    #4 0x7ff9cb1bf7ac in WriteBuffer<unsigned int> /builds/worker/checkouts/gecko/image/SurfacePipe.h:300:5
    #5 0x7ff9cb1bf7ac in mozilla::image::WriteState mozilla::image::SurfacePipe::WriteBuffer<unsigned int>(unsigned int const*) /builds/worker/checkouts/gecko/image/SurfacePipe.h:707:19
    #6 0x7ff9cb1b6705 in mozilla::image::nsAVIFDecoder::Decode(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*) /builds/worker/checkouts/gecko/image/decoders/nsAVIFDecoder.cpp:1729:31
    #7 0x7ff9cb1b2d6e in mozilla::image::nsAVIFDecoder::DoDecode(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*) /builds/worker/checkouts/gecko/image/decoders/nsAVIFDecoder.cpp:1181:25
    #8 0x7ff9cb09df4f in mozilla::image::Decoder::Decode(mozilla::image::IResumable*) /builds/worker/checkouts/gecko/image/Decoder.cpp:177:19
    #9 0x7ff9cb09d887 in mozilla::image::AnimationSurfaceProvider::Run() /builds/worker/checkouts/gecko/image/AnimationSurfaceProvider.cpp:232:36
    #10 0x7ff9cb0d84bb in mozilla::image::DecodingTask::Run() /builds/worker/checkouts/gecko/image/DecodePool.cpp:146:12
    #11 0x7ff9c849234d in mozilla::TaskController::RunPoolThread() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:328:33
    #12 0x7ff9ea8b6628 in _pt_root /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:201:5
    #13 0x7ff9ead19b42 in start_thread nptl/pthread_create.c:442:8
    #14 0x7ff9eadab9ff  misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

0x62e00002e043 is located 3 bytes to the right of 40000-byte region [0x62e000024400,0x62e00002e040)
allocated by thread T21 here:
    #0 0x561914d518ee in malloc /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3
    #1 0x561914d95055 in moz_xmalloc /builds/worker/checkouts/gecko/memory/mozalloc/mozalloc.cpp:52:15
    #2 0x7ff9cb1af0f6 in operator new[] /builds/worker/workspace/obj-build/dist/include/mozilla/cxxalloc.h:42:10
    #3 0x7ff9cb1af0f6 in mozilla::detail::UniqueSelector<unsigned char []>::UnknownBound mozilla::MakeUnique<unsigned char []>(unsigned long) /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:612:23
    #4 0x7ff9cb1b5d05 in mozilla::image::nsAVIFDecoder::Decode(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*) /builds/worker/checkouts/gecko/image/decoders/nsAVIFDecoder.cpp:1662:33
    #5 0x7ff9cb1b2d6e in mozilla::image::nsAVIFDecoder::DoDecode(mozilla::image::SourceBufferIterator&, mozilla::image::IResumable*) /builds/worker/checkouts/gecko/image/decoders/nsAVIFDecoder.cpp:1181:25
    #6 0x7ff9cb09df4f in mozilla::image::Decoder::Decode(mozilla::image::IResumable*) /builds/worker/checkouts/gecko/image/Decoder.cpp:177:19
    #7 0x7ff9cb09d887 in mozilla::image::AnimationSurfaceProvider::Run() /builds/worker/checkouts/gecko/image/AnimationSurfaceProvider.cpp:232:36
    #8 0x7ff9cb0d84bb in mozilla::image::DecodingTask::Run() /builds/worker/checkouts/gecko/image/DecodePool.cpp:146:12
    #9 0x7ff9c849234d in mozilla::TaskController::RunPoolThread() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:328:33
    #10 0x7ff9ea8b6628 in _pt_root /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:201:5
    #11 0x7ff9ead19b42 in start_thread nptl/pthread_create.c:442:8

Thread T21 created by T0 (Isolated Web Co) here:
    #0 0x561914d3a7dc in pthread_create /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_interceptors.cpp:208:3
    #1 0x7ff9ea8a66f9 in _PR_CreateThread /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:458:14
    #2 0x7ff9ea897b6e in PR_CreateThread /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:533:12
    #3 0x7ff9c8492feb in mozilla::TaskController::InitializeThreadPool() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:227:10
    #4 0x7ff9c8494e76 in mozilla::TaskController::AddTask(already_AddRefed<mozilla::Task>&&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:394:7
    #5 0x7ff9c9f3ddc5 in DispatchOffThreadTask(JS::DispatchReason) /builds/worker/checkouts/gecko/js/xpconnect/src/XPCJSContext.cpp:1179:26
    #6 0x7ff9d6534a35 in js::GlobalHelperThreadState::submitTask(js::GCParallelTask*, js::AutoLockHelperThreadState const&) /builds/worker/checkouts/gecko/js/src/vm/HelperThreads.cpp:2302:3
    #7 0x7ff9d7001ada in startWithLockHeld /builds/worker/checkouts/gecko/js/src/gc/GCParallelTask.cpp:44:23
    #8 0x7ff9d7001ada in js::GCParallelTask::startOrRunIfIdle(js::AutoLockHelperThreadState&) /builds/worker/checkouts/gecko/js/src/gc/GCParallelTask.cpp:72:3
    #9 0x7ff9d706c93d in js::gc::GCRuntime::startBackgroundFree() /builds/worker/checkouts/gecko/js/src/gc/Sweeping.cpp:448:12
    #10 0x7ff9d6fd87d5 in js::gc::GCRuntime::collectNursery(JS::GCOptions, JS::GCReason, js::gcstats::PhaseKind) /builds/worker/checkouts/gecko/js/src/gc/GC.cpp:4589:3
    #11 0x7ff9d6fe004c in js::gc::GCRuntime::minorGC(JS::GCReason, js::gcstats::PhaseKind) /builds/worker/checkouts/gecko/js/src/gc/GC.cpp:4558:3
    #12 0x7ff9d6fa40ff in JSObject* js::gc::GCRuntime::tryNewNurseryObject<(js::AllowGC)1>(JSContext*, unsigned long, unsigned long, JSClass const*, js::gc::AllocSite*) /builds/worker/checkouts/gecko/js/src/gc/Allocator.cpp:109:23
    #13 0x7ff9d6fa3e76 in JSObject* js::gc::detail::AllocateObject<(js::AllowGC)1>(JSContext*, js::gc::AllocKind, unsigned long, js::gc::InitialHeap, JSClass const*, js::gc::AllocSite*) /builds/worker/checkouts/gecko/js/src/gc/Allocator.cpp:65:28
    #14 0x7ff9d63757b7 in js::NativeObject::create(JSContext*, js::gc::AllocKind, js::gc::InitialHeap, JS::Handle<js::SharedShape*>, js::gc::AllocSite*) /builds/worker/checkouts/gecko/js/src/vm/NativeObject-inl.h:446:11
    #15 0x7ff9d6375314 in js::NewPlainObjectBaselineFallback(JSContext*, JS::Handle<js::SharedShape*>, js::gc::AllocKind, js::gc::AllocSite*) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:5297:10
    #16 0x21aa68c96b82  (<unknown module>)
    #17 0x21aa68c9b4eb  (<unknown module>)
    #18 0x21aa68ca385d  (<unknown module>)
    #19 0x21aa68c9c53b  (<unknown module>)
    #20 0x21aa68ca385d  (<unknown module>)
    #21 0x21aa68c9c62f  (<unknown module>)
    #22 0x21aa68c924ed  (<unknown module>)
    #23 0x7ff9d78cfd74 in EnterJit /builds/worker/checkouts/gecko/js/src/jit/Jit.cpp:104:5
    #24 0x7ff9d78cfd74 in js::jit::MaybeEnterJit(JSContext*, js::RunState&) /builds/worker/checkouts/gecko/js/src/jit/Jit.cpp:205:10
    #25 0x7ff9d63362ec in js::RunScript(JSContext*, js::RunState&) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:421:32
    #26 0x7ff9d6363330 in js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct, js::CallReason) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:579:13
    #27 0x7ff9d6364fef in InternalCall /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:614:10
    #28 0x7ff9d6364fef in js::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, js::AnyInvokeArgs const&, JS::MutableHandle<JS::Value>, js::CallReason) /builds/worker/checkouts/gecko/js/src/vm/Interpreter.cpp:646:8
    #29 0x7ff9d646e28d in JS::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, JS::HandleValueArray const&, JS::MutableHandle<JS::Value>) /builds/worker/checkouts/gecko/js/src/vm/CallAndConstruct.cpp:117:10
    #30 0x7ff9cc19e2d1 in mozilla::dom::MessageListener::ReceiveMessage(mozilla::dom::BindingCallContext&, JS::Handle<JS::Value>, mozilla::dom::ReceiveMessageArgument const&, JS::MutableHandle<JS::Value>, mozilla::ErrorResult&) /builds/worker/workspace/obj-build/dom/bindings/MessageManagerBinding.cpp:6028:8
    #31 0x7ff9cb41820b in void mozilla::dom::MessageListener::ReceiveMessage<JS::Rooted<JS::Value>>(JS::Rooted<JS::Value> const&, mozilla::dom::ReceiveMessageArgument const&, JS::MutableHandle<JS::Value>, mozilla::ErrorResult&, char const*, mozilla::dom::CallbackObject::ExceptionHandling, JS::Realm*) /builds/worker/workspace/obj-build/dist/include/mozilla/dom/MessageManagerBinding.h:639:12
    #32 0x7ff9cb415a22 in nsFrameMessageManager::ReceiveMessage(nsISupports*, nsFrameLoader*, bool, nsTSubstring<char16_t> const&, bool, mozilla::dom::ipc::StructuredCloneData*, nsTArray<mozilla::dom::ipc::StructuredCloneData>*, mozilla::ErrorResult&) /builds/worker/checkouts/gecko/dom/base/nsFrameMessageManager.cpp:746:25
    #33 0x7ff9cb41fac3 in nsFrameMessageManager::ReceiveMessage(nsISupports*, nsFrameLoader*, nsTSubstring<char16_t> const&, bool, mozilla::dom::ipc::StructuredCloneData*, nsTArray<mozilla::dom::ipc::StructuredCloneData>*, mozilla::ErrorResult&) /builds/worker/checkouts/gecko/dom/base/nsFrameMessageManager.h:167:5
    #34 0x7ff9d011dac6 in mozilla::dom::ContentChild::RecvAsyncMessage(nsTString<char16_t> const&, mozilla::dom::ClonedMessageData const&) /builds/worker/checkouts/gecko/dom/ipc/ContentChild.cpp:2368:10
    #35 0x7ff9d03fd6a1 in mozilla::dom::PContentChild::OnMessageReceived(IPC::Message const&) /builds/worker/workspace/obj-build/ipc/ipdl/PContentChild.cpp:15539:80
    #36 0x7ff9c9c51bc9 in mozilla::ipc::MessageChannel::DispatchAsyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&) /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:1800:25
    #37 0x7ff9c9c4ed1d in mozilla::ipc::MessageChannel::DispatchMessage(mozilla::ipc::ActorLifecycleProxy*, mozilla::UniquePtr<IPC::Message, mozilla::DefaultDelete<IPC::Message>>) /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:1725:9
    #38 0x7ff9c9c4f8ee in mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::ActorLifecycleProxy*, mozilla::ipc::MessageChannel::MessageTask&) /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:1525:3
    #39 0x7ff9c9c50b1e in mozilla::ipc::MessageChannel::MessageTask::Run() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:1623:14
    #40 0x7ff9c84a2159 in mozilla::RunnableTask::Run() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:539:16
    #41 0x7ff9c84985f7 in mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:852:26
    #42 0x7ff9c8495878 in mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:684:15
    #43 0x7ff9c8495fa0 in mozilla::TaskController::ProcessPendingMTTask(bool) /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:462:36
    #44 0x7ff9c84a8631 in operator() /builds/worker/checkouts/gecko/xpcom/threads/TaskController.cpp:188:37
    #45 0x7ff9c84a8631 in mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_2>::Run() /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:546:5
    #46 0x7ff9c84cb815 in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1197:16
    #47 0x7ff9c84d5c64 in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:477:10
    #48 0x7ff9c9c597ce in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:85:21
    #49 0x7ff9c9ad8547 in RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:381:10
    #50 0x7ff9c9ad8547 in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:374:3
    #51 0x7ff9c9ad8547 in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:356:3
    #52 0x7ff9d0f855a9 in nsBaseAppShell::Run() /builds/worker/checkouts/gecko/widget/nsBaseAppShell.cpp:148:27
    #53 0x7ff9d5f45e88 in XRE_RunAppShell() /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:742:20
    #54 0x7ff9c9ad8547 in RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:381:10
    #55 0x7ff9c9ad8547 in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:374:3
    #56 0x7ff9c9ad8547 in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:356:3
    #57 0x7ff9d5f4561f in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:675:34
    #58 0x561914d8e494 in content_process_main(mozilla::Bootstrap*, int, char**) /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:57:28
    #59 0x561914d8e957 in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:353:18
    #60 0x7ff9eacaed8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

Looks like this is caused by allowing an upscaled render rect to be sent to the surface pipe. My understanding is that the proper solution is to error out when the render rect's size doesn't fit within the decoded size, since the pipe is only able to downscale.

I have a patch for this, but I'm unsure based on the security bug guidelines whether I should merge this into another patch or otherwise obfuscate it. This should only affect a very small subset of Nightly users, so it seems fine to submit the patch as normal, no?

I think you can just submit the patch to this bug, this bug is hidden. If we need to do something more complicated we can figure it out.

I needed image.avif.sequence.enabled=true in order to reproduce this.

Bug 1814677 made me realize that I was using SurfacePipeFactory::CreateSurfacePipe the wrong way, the aFrameRect should always be equal to the total decoded frame rectangle rather than the render rectangle. The patch here will be obsoleted in favor of fixing that bug and then this one can be closed.

Resolving, as the fix for the issue mentioned in comment #6 fixes this issue as well.